{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_3xxr5fb",
    "id": "B116BFDF0D464FF49A85A582357D0B4D",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# 线性回归\n",
    "\n",
    "主要内容包括：\n",
    "\n",
    "1. 线性回归的基本要素\n",
    "2. 线性回归模型从零开始的实现\n",
    "3. 线性回归模型使用 Tensorflow 的简洁实现\n",
    "\n",
    "代码基于 `TensorFlow 2.0`。\n",
    "\n",
    "\n",
    "\n",
    "线性回归输出是一个连续值，因此适用于回归问题。回归问题在实际中很常见，如预测房屋价格、气温、销售额等连续值的问题。与回归问题不同，分类问题中模型的最终输出是一个离散值。我们所说的图像分类、垃圾邮件识别、疾病检测等输出为离散值的问题都属于分类问题的范畴。softmax回归则适用于分类问题。\n",
    "\n",
    "由于线性回归和softmax回归都是单层神经网络，它们涉及的概念和技术同样适用于大多数的深度学习模型。我们首先以线性回归为例，介绍大多数深度学习模型的基本要素和表示方法。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_ht8ukap",
    "id": "8FCA1BC77B7F479BA1398473C2691BB0",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 线性回归的基本要素\n",
    "\n",
    "### 模型\n",
    "设房屋的面积为 $x_1$，房龄为 $x_2$，售出价格为 $y$。我们需要建立基于输入 $x_1$ 和 $x_2$ 来计算输出 $y$ 的表达式，也就是模型（model）。顾名思义，线性回归假设输出与各个输入之间是线性关系：\n",
    "$$\n",
    "\\hat{y} = x_1 w_1 + x_2 w_2 + b\n",
    "$$\n",
    "其中 $w_1$ 和 $w_2$ 是权重（weight），$b$ 是偏差（bias），且均为标量。它们是线性回归模型的参数（parameter）。模型输出 $\\hat{y}$ 是线性回归对真实价格 $y$ 的预测或估计。我们通常允许它们之间有一定误差。\n",
    "\n",
    "\n",
    "\n",
    "### 数据集\n",
    "我们通常收集一系列的真实数据，例如多栋房屋的真实售出价格和它们对应的面积和房龄。我们希望在这个数据上面寻找模型参数来使模型的预测价格与真实价格的误差最小。在机器学习术语里，该数据集被称为训练数据集（training data set）或训练集（training set），一栋房屋被称为一个样本（sample），其真实售出价格叫作标签（label），用来预测标签的两个因素叫作特征（feature）。特征用来表征样本的特点。\n",
    "### 损失函数\n",
    "在模型训练中，我们需要衡量价格预测值与真实值之间的误差。通常我们会选取一个非负数作为误差，且数值越小表示误差越小。一个常用的选择是平方孙树函数。 它在评估索引为 $i$ 的样本误差的表达式为\n",
    "\n",
    "\n",
    "$$\n",
    "l^{(i)}(\\mathbf{w}, b) = \\frac{1}{2} \\left(\\hat{y}^{(i)} - y^{(i)}\\right)^2,\n",
    "$$\n",
    "\n",
    "\n",
    "\n",
    "$$\n",
    "L(\\mathbf{w}, b) =\\frac{1}{n}\\sum_{i=1}^n l^{(i)}(\\mathbf{w}, b) =\\frac{1}{n} \\sum_{i=1}^n \\frac{1}{2}\\left(\\mathbf{w}^\\top \\mathbf{x}^{(i)} + b - y^{(i)}\\right)^2.\n",
    "$$\n",
    "\n",
    "在模型训练中，我们希望找出一组模型参数，记为 $w_1^*, w_2^*, b^*$，来使训练样本平均损失最小：\n",
    "$$\n",
    "w_1^*, w_2^*, b^* = \\underset{w_1, w_2, b}{\\arg\\min} \\ell(w_1, w_2, b)\n",
    "$$\n",
    "\n",
    "### 优化函数 - 随机梯度下降\n",
    "当模型和损失函数形式较为简单时，上面的误差最小化问题的解可以直接用公式表达出来。这类解叫作解析解（analytical solution）。本节使用的线性回归和平方误差刚好属于这个范畴。然而，大多数深度学习模型并没有解析解，只能通过优化算法有限次迭代模型参数来尽可能降低损失函数的值。这类解叫作数值解（numerical solution）。\n",
    "\n",
    "在求数值解的优化算法中，小批量随机梯度下降（mini-batch stochastic gradient descent）在深度学习中被广泛使用。它的算法很简单：先选取一组模型参数的初始值，如随机选取；接下来对参数进行多次迭代，使每次迭代都可能降低损失函数的值。在每次迭代中，先随机均匀采样一个由固定数目训练数据样本所组成的小批量（mini-batch）$\\mathcal{B}$，然后求小批量中数据样本的平均损失有关模型参数的导数（梯度），最后用此结果与预先设定的一个正数的乘积作为模型参数在本次迭代的减小量。   \n",
    "\n",
    "$$\n",
    "(\\mathbf{w},b) \\leftarrow (\\mathbf{w},b) - \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}} \\partial_{(\\mathbf{w},b)} l^{(i)}(\\mathbf{w},b)\n",
    "$$\n",
    "  \n",
    "  \n",
    " 在训练本节讨论的线性回归模型的过程中，模型的每个参数将作如下迭代：\n",
    "$$\n",
    "\\begin{aligned}\n",
    "w_1 &\\leftarrow w_1 -   \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}} \\frac{ \\partial \\ell^{(i)}(w_1, w_2, b)  }{\\partial w_1} = w_1 -   \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}}x_1^{(i)} \\left(x_1^{(i)} w_1 + x_2^{(i)} w_2 + b - y^{(i)}\\right),\\\\\n",
    "w_2 &\\leftarrow w_2 -   \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}} \\frac{ \\partial \\ell^{(i)}(w_1, w_2, b)  }{\\partial w_2} = w_2 -   \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}}x_2^{(i)} \\left(x_1^{(i)} w_1 + x_2^{(i)} w_2 + b - y^{(i)}\\right),\\\\\n",
    "b &\\leftarrow b -   \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}} \\frac{ \\partial \\ell^{(i)}(w_1, w_2, b)  }{\\partial b} = b -   \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}}\\left(x_1^{(i)} w_1 + x_2^{(i)} w_2 + b - y^{(i)}\\right).\n",
    "\\end{aligned}\n",
    "$$\n",
    "学习率: $\\eta$代表在每次优化中，能够学习的步长的大小    \n",
    "批量大小: $\\mathcal{B}$是小批量计算中的批量大小batch size   \n",
    "\n",
    "总结一下，优化函数的有以下两个步骤：\n",
    "\n",
    "- (i)初始化模型参数，一般来说使用随机初始化；\n",
    "- (ii)我们在数据上迭代多次，通过在负梯度方向移动参数来更新每个参数。\n",
    "\n",
    "### 神经网络表示\n",
    "\n",
    "\n",
    "在深度学习中，我们可以使用神经网络图直观地表现模型结构。为了更清晰地展示线性回归作为神经网络的结构，图1使用神经网络图表示本节中介绍的线性回归模型。神经网络图隐去了模型参数权重和偏差。\n",
    "\n",
    "<div align=center>\n",
    "<img width=\"250\" src=\"../img/chapter03/3.1_linreg.svg\"/>\n",
    "</div>\n",
    "<div align=center>图1 线性回归是一个单层神经网络</div>\n",
    "\n",
    "在图1所示的神经网络中，输入分别为 $x_1$ 和 $x_2$，因此输入层的输入个数为2。输入个数也叫特征数或特征向量维度。图1中网络的输出为 $o$，输出层的输出个数为1。需要注意的是，我们直接将图1中神经网络的输出 $o$ 作为线性回归的输出，即 $\\hat{y} = o$。由于输入层并不涉及计算，按照惯例，图1所示的神经网络的层数为1。所以，线性回归是一个单层神经网络。输出层中负责计算 $o$ 的单元又叫神经元。在线性回归中，$o$ 的计算依赖于 $x_1$ 和 $x_2$。也就是说，输出层中的神经元和输入层中各个输入完全连接。因此，这里的输出层又叫全连接层（fully-connected layer）或稠密层（dense layer）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_v3gyr0b",
    "id": "469D697FF90B48B7B0B61AED429EB8D6",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 矢量计算\n",
    "在模型训练或预测时，我们常常会同时处理多个数据样本并用到矢量计算。在介绍线性回归的矢量计算表达式之前，让我们先考虑对两个向量相加的两种方法。\n",
    "\n",
    "\n",
    "1. 向量相加的一种方法是，将这两个向量按元素逐一做标量加法。\n",
    "2. 向量相加的另一种方法是，将这两个向量直接做矢量加法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T15:02:51.347342Z",
     "start_time": "2020-03-26T15:02:22.033068Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.1.0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from time import time\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T15:02:52.042527Z",
     "start_time": "2020-03-26T15:02:52.038509Z"
    }
   },
   "source": [
    "定义下面两个 1000 维的向量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T15:02:55.117160Z",
     "start_time": "2020-03-26T15:02:53.693549Z"
    },
    "graffitiCellId": "id_bp6luds",
    "id": "631AD2C3EA1A431287E30A95D535D877",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "a = tf.ones((1000,))\n",
    "b = tf.ones((1000,))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_g9h7dg8",
    "id": "2698821CF46844989522D09B8B1C76DB",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "现在我们可以来测试了。首先将两个向量使用for循环按元素逐一做标量加法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T15:04:25.236368Z",
     "start_time": "2020-03-26T15:04:24.948121Z"
    },
    "graffitiCellId": "id_eoz706b",
    "id": "DF2AACFBA2EA42698CC82C33AF79AEDB",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.28026413917541504"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "start = time()\n",
    "c = tf.Variable(tf.zeros((1000,)))\n",
    "for i in range(1000):\n",
    "    c[i].assign(a[i] + b[i])\n",
    "\n",
    "time() - start"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_akkwkh8",
    "id": "B00F06B72BB5471DA82C945B04FED140",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "向量相加的另一种方法是，将这两个向量直接做矢量加法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T15:04:58.297453Z",
     "start_time": "2020-03-26T15:04:58.278934Z"
    },
    "graffitiCellId": "id_a8sw68j",
    "id": "6D2503874A514A7590AF8F710B5F325C",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.014528751373291016"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "start = time()\n",
    "c.assign(a + b)\n",
    "time() - start"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_oonn3xx",
    "id": "B0CA3D998E0A4B5C848F9C1BAC37DB13",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "结果很明显,后者比前者运算速度更快。因此，我们应该尽可能采用矢量计算，以提升计算效率。\n",
    "\n",
    "让我们再次回到本节的房价预测问题。如果我们对训练数据集里的3个房屋样本（索引分别为1、2和3）逐一预测价格，将得到\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\hat{y}^{(1)} &= x_1^{(1)} w_1 + x_2^{(1)} w_2 + b,\\\\\n",
    "\\hat{y}^{(2)} &= x_1^{(2)} w_1 + x_2^{(2)} w_2 + b,\\\\\n",
    "\\hat{y}^{(3)} &= x_1^{(3)} w_1 + x_2^{(3)} w_2 + b.\n",
    "\\end{aligned}\n",
    "$$\n",
    "现在，我们将上面3个等式转化成矢量计算。设\n",
    "$$\n",
    "\\boldsymbol{\\hat{y}} =\n",
    "\\begin{bmatrix}\n",
    "    \\hat{y}^{(1)} \\\\\n",
    "    \\hat{y}^{(2)} \\\\\n",
    "    \\hat{y}^{(3)}\n",
    "\\end{bmatrix},\\quad\n",
    "\\boldsymbol{X} =\n",
    "\\begin{bmatrix}\n",
    "    x_1^{(1)} & x_2^{(1)} \\\\\n",
    "    x_1^{(2)} & x_2^{(2)} \\\\\n",
    "    x_1^{(3)} & x_2^{(3)}\n",
    "\\end{bmatrix},\\quad\n",
    "\\boldsymbol{w} =\n",
    "\\begin{bmatrix}\n",
    "    w_1 \\\\\n",
    "    w_2\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "对3个房屋样本预测价格的矢量计算表达式为$\\boldsymbol{\\hat{y}} = \\boldsymbol{X} \\boldsymbol{w} + b,$ 其中的加法运算使用了广播机制。例如："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T15:08:07.202943Z",
     "start_time": "2020-03-26T15:08:07.184019Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3,), dtype=float32, numpy=array([11., 11., 11.], dtype=float32)>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = tf.ones((3,))\n",
    "b = 10\n",
    "a + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_3y8h3t7",
    "id": "84D91561397548D7ACB5FAB71E66AB9B",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 线性回归模型从零开始的实现\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T15:09:14.940969Z",
     "start_time": "2020-03-26T15:09:09.583948Z"
    },
    "graffitiCellId": "id_3snj2zc",
    "id": "B3148881D9514B898929430997FD781C",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.1.0\n"
     ]
    }
   ],
   "source": [
    "# import packages and modules\n",
    "%matplotlib inline\n",
    "import tensorflow as tf\n",
    "from IPython import display\n",
    "from matplotlib import pyplot as plt\n",
    "import numpy as np\n",
    "import random\n",
    "\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_ofruiuq",
    "id": "D7C96AC35B12411E8A1530B965CB34E0",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 生成数据集\n",
    "使用线性模型来生成数据集，生成一个1000个样本的数据集，下面是用来生成数据的线性关系：\n",
    "\n",
    "$$\n",
    "\\mathrm{price} = w_{\\mathrm{area}} \\cdot \\mathrm{area} + w_{\\mathrm{age}} \\cdot \\mathrm{age} + b\n",
    "$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T16:04:01.040990Z",
     "start_time": "2020-03-26T16:04:01.035972Z"
    },
    "graffitiCellId": "id_h3bosrm",
    "id": "1A5F9ED7F99643A3A440960077439F0F",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 特征数\n",
    "num_inputs = 2\n",
    "# 样本数\n",
    "num_examples = 1000\n",
    "# 权重 w 向量\n",
    "true_w = [2, -3.4]\n",
    "# 偏差\n",
    "true_b = 4.2\n",
    "\n",
    "features = tf.random.normal((num_examples, num_inputs), stddev=1)\n",
    "labels = true_w[0] * features[:,0] + true_w[1] * features[:, 1] + true_b\n",
    "# 添加随机噪声\n",
    "labels += tf.random.normal(labels.shape, stddev=0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T15:21:20.372045Z",
     "start_time": "2020-03-26T15:21:20.367089Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([ 0.7026267 -0.5950136], shape=(2,), dtype=float32) tf.Tensor(7.6275563, shape=(), dtype=float32)\n",
      "(1000, 2) (1000,)\n"
     ]
    }
   ],
   "source": [
    "print(features[0], labels[0])\n",
    "print(features.shape,labels.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_gr10soh",
    "id": "937B9B59AC2343B58488AAA9B7C11C2A",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 使用图像来展示生成的数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T15:14:05.237082Z",
     "start_time": "2020-03-26T15:14:05.126375Z"
    },
    "graffitiCellId": "id_ov2af2a",
    "id": "8E2E1E16060241C6A33E4CF1EC65DF1D",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO2de3BUV37nv4eHHD0ISEIomJdAakMUL9YM7cdiHgOC7DhhcbI1sJnslpXHLnalxqt4vFMTJ844k3HKSSa2Q01txWY3TjS7WWfMZDLj9cSJDWYwMsG2mMIM1iDUQsJgE9FqCcZSEzcSZ/+4fa7OvX1v9+133+7vp4qS6L6PX7fN9/zu7/weQkoJQggh/mVOsQ0ghBCSHRRyQgjxORRyQgjxORRyQgjxORRyQgjxOfOKcdPFixfLlpaWYtyaEEJ8y8mTJ8eklE3214si5C0tLejr6yvGrQkhxLcIIS44vc7QCiGE+BwKOSGE+BwKOSGE+BwKOSGE+BwKOSGE+BwKOSGE+BwKOSGE+JyyFfLxqRiePzqE8alYsU0hhJC8UrZCfrDvIp569SwO9l0stimEEJJXilLZWQj2BFdYfhJCSLlStkLeUFuFB7e2FtsMQgjJO2UbWiGEkEqBQk4IIT6HQk4IIT6HQk4IIT6HQu5TmCdPCFFQyH0K8+QJIYqyTT8sd5gnTwhRUMh9CvPkCSEKhlYIIcTnUMgJIcTnUMgJIcTnUMgJIcTnUMgJIcTnUMgJIcTnUMgJIcTnUMgJIcTnUMgJIcTn+ErI2SiKEEIS8ZWQs1EUIYQk4qteK2wURQghiXj2yIUQLwghrgghzmiv/YEQ4kMhxKn4n1/Ij5kGqlFUQ21VPm9DCCG+Ip3Qyl8D+KzD689KKTvif/4hN2ZVJtwDIIRkgmchl1K+CWA8j7ZUPNwDIIRkQi5i5F8QQjwAoA/Ao1LKCaeDhBD7AOwDgJUrV+bgtuUH9wAIIZmQbdbKXwBoBdAB4DKAp90OlFIekFIGpZTBpqamLG9bnnAPgBCSCVkJuZRyVEo5I6W8CeB/ArgrN2YRQgjxSlZCLoRYqv31lwGccTuW5AZuiBJC7KSTfvgigH8GsFYIcUkI8ZsA/lQI8SMhxGkA2wA8kic7Kxa7cKe7IUrhJ6T88bzZKaX8vMPLf5lDW4gDSrgB4MGtrWlviNrPJ4SUH76q7KxE7MKtNkQzPZ8QUn74qtdKJZJpJosKqQBgJgwhZQ6FPA+UQlw6WSy9FOwjhOQOhlZyzPhUDI++dApHBsIAiheXThZSYdyckPKCQp5jDvZdxJGBMLatbSpqXDpZLJ1xc0LKC4ZWssQeptgTXIHH7luHp/d2ZBzX9hLyyCY8wgpSQsoLCnmW2GPR2YhkOjniPceH8dSrZ9FzfNh8jbFvQioThlayJJdhinSudf3GTctPgLFvQioVeuRpYvd6nTxwJ89YvTYUnnT1mtPx5qvnz7H8BGbDOox9E1JZ0CNPEy9er9Mx6rUT5yM5yWjp2rgaNVXzLKKdbrEQYCwwB/suYk9wBWPmhPgUCnmaeAl/OB2jft/R3ox71owmnJ+uoDqJdiaizHAMIf6HQp4mXrxep2P011q31iWckwtB9XoNXfCZikiI/6GQlwiZCKrdA/d6Dbvg66LPUAsh/oNCXiJkEt+2C7LXayQT/J7jw9h/OIRobBqP7Fyblj2EkOJAIfcxyQQ5mWedXPCF7SchpNShkPuYZIKcacy9a2MLaqrmFiRmzjAOIbmBQp5niiVWmW5iZhLiyRRmzBCSGyjkeaYQYuW0WBRSkDOFGTOE5AYKeZ4phFj5ybO1Lzqlbi8hfoBCnmcKIVZ+8mz9tOgQ4hco5GVAoTzbXMT7/bToEOIX2DSrBClEO9pM7uGlzW6q67IXOiG5h0JeIugCmE5fcrdrpMLpHuNTMTz7+gCeff2c4zVUd8Ud7c2u98nUdkJI5jC0UiLo3REf39UOIP3wQzq9VqKxGXR3tmFPcIW5eERj09h/OAQAZi65HkpR3vTzR4dc76M3B3v+6BBzxAkpAPTIS4Q9wRXYtrYJRwbCONQ/mlH4wWs/8oN9F7H/8CBqqox1/NGXTsWFWWDf5jXY1NaIHe3Nrt51svsosT/UP5qxZ85JR4SkBz3yEqGhtiruifdjR3tzxtfw0gBL33BUw6I3tTUCkKiumoPeUASH+kddNya9bK5ms6nJzBZC0oMeeQlxqH/U9MhzgVscXBf3He3N2La2Ce23LoyHVYTpbWezMZnNubrHT++ckNTQIy8h7F5stul+Tl6x3dtVi8f65QstAm4nl60GUl1L9/iTxeMJIQb0yEuYbDNAVI/yg30XMT4VS9jkBGB65Ls7llnE0+4BJ7PF7jWn8qJTfS79/D3BFejuDCAam6ZXTogL9MhLCLu3nCrObPdsnTxd/ZoAsP/wILo728zjlEd+z5pRtG6tc41PO9kym+0yg/2HB81zUsW4U30u+/k1VXPx1KtnUVM1L2OvnJ0WSTnjWciFEC8A2AXgipTy9vhrDQC+BaAFwAiAvVLKidybWRnYBS7VpqJd8JwE1Ek0o7EZ8zj7++lscKr7dXe2WbJYUgl1qs/l1aZ04AYqKWeElNLbgUJsATAJ4JuakP8pgHEp5R8LIX4HQL2U8suprhUMBmVfX18WZlcGqbxILx55Jtf1es74VAw9x0cASHRtXO3pWvnwjL1c07B1GIBA18YWeuXElwghTkopg/bXPcfIpZRvAhi3vXw/gJ747z0AfiljC0kCPceH8dSrZ+MClIg9M8RrpojTcZnEtRtqq1BTNRf7D4c8x/HzUfnp5ZqGrfOw//Agq05J2ZFtjLxZSnkZAKSUl4UQS9wOFELsA7APAFauXJnlbSuF3I1dc/Na7VWd0dg0aqrmOeaeR2Mz5qajes/tdTfy0TTL6zXZsIuUKwXLWpFSHpBSBqWUwaampkLd1td0bWzBY/etQ9fGlrTPtXvYbl7rbOzYyB8HBJ569SwefemUxTt38769euXKHgCenhrSyR/P5kmEkHIgW498VAixNO6NLwVwJRdGEYNM29OOT8Xw6EuncGQgDMA9A8ZIR5xGd2fAjBuPT8Vw+tJVHBkI42DfxYTMlWhsGtHYTIJXbr+2ur56Ckh3s5Gbk4R4J1uP/GUAXfHfuwB8L8vrVRS5rFq0d088MhDGtrVNCRkwujdq9FwJoaZqriXO/vTeDsdeKm5xZjdPV4/xq3z1VO0H1OfY0d7sqW8MISS99MMXAXwGwGIhxCUATwD4YwAvCSF+E8AHAPbkw8hyxc3rzCSzQ7+W7iE31FZhKDyJJ1/px+O72tHaVGeek0kvlXTizNdv3DR/2vPV3eg5PmzG6h/ZuRYAc8AJSYVnIZdSft7lrc4c2VJxuIliJmEFu3jr5z35Sn88zNKPv/r1uyzCmKrdrdehzo5iq1JbpUy5KarOV+J/8sKEeZyfwixcdEgxYIl+EXELSXhtR5vqWipM8fD2ADa1NSKwZEFagytSpT/qOF2zOt4mt7pqXspNUXV+9fw52La2Cb2hiHmcl++jVJprcbAGKQYs0S9BnLzebMItj923DpsDTXjq1bNorKtKGh7R75NO+qOTxz2bbSPNvilu97W/N2uDN/TBHE/v7UjLG86lF80UR1IMKOQ+Idtwi/5ashi4uk80Ng0AZkZLKpTHrfdEcXot2QKlv6f/nuyzq/N3tDfjxPmIY7ZNKnIZukk304ihGJILKOQ+IRNPzy4qXgRGXV81wjJyy2GObQPgKjxuC0cyu3uOj2D/4UFEYzN4ZOdt5uu6wCW7hi7Cj+9qR2z6DCLxTo/JbHWyuxjj6fwU/yelC4XcJ2SaU+5EMi9Q3Wd8KmaZ26l3UHQTHicb9es5i6Q0f+p22bNXkmXRqNz2l099iLeGInhrKIL+j36CDasWmTNIk313ysZnXz+XsKjk22NmKIbkAgp5BeLFC9RFOZmnnQq3VreKro2rzZYA1gXDW3xe5bYbXRgDuLe1EW8NRdAbGkP7rT+NTW2LEZn8xFP7AKdFxc3uXJHLBZpULhTyMiSVF+nFC7RfQ4lNulkhbq1uFW4LxkTUqDDdeltTynCH8soB4A9/6Xa8fOpDGAuARG9oDL2hMTTW3ZJSMJ0WFbvdjGmTUoRCXoak8ri9eIFuWSD2a6cSNrUJubtjGVqb6jA+FcOzr5+DU+tb3a6e4yM4MhDGjRlDjFNlo5y+dA1HBsKoqZprKSRSgu7lCcJtUbFXwzKmTUoNCrmPcRNRt74q6XiSe4IrLFkge4Ir8NzRIfSNjOPu1fVmqX0qYXOaQKRCFQAsnRb1DJS+EaNjcmx6BpvaFlvssPdAd2pJABjCrG+gpoPbYseYNilFKOQ+xk1Ek03z8ZpnrXqu6E2vDrx53nz/UL8hzMrjVsJuXzD0zcih8CSisRns27wG1VVzoDotKvuVjS/1XcRQeAqtTbV4Z2QC3Z1t2LBqEaKxGTPLBTC88Kf3drh6z15JZ0AHY9qkFGFlp49JpwJ0T3AFtq1tMj1bL+jVonuCK7BvyxrcvboB+zavNu+pPO5D/aMAZhcM1QZXb7T1le+ewf7Dg6iumoNHdq412/TqQryprRFD4Sm0NNbg65+7I97Gd7V5DUBi35Y1WNVQY34WtwpZr9We9mpMVmcSv0GPPMcUcjMslXdot0X3sJOd4zYSrbG2Cn/xnzckDePsaG/GS/FQR8/xYdRUzTO99chUDG8NRaAyUez2N9RWYc3iOvSGIhiJRPHuyDge3NqKH16YwIvvfIAH7lmF3R3L8OQr/bgwHjW7KbpthnqNZ9s/A8MnxG9QyHNMKW2G2W3xusmpcq8BOOaS6xks9kXrUP8ohsJT2La2CfbQyfhUDI21VUnF90cfXgUA3Lrwp0wh/eJLpzASiQIIY1l9tRkPVwuT2/dtD/u44bSgFPu/HSHpQCHPMaXkzWViy2wqn5Ht4dQaV+Ekok7H6f1X9MKbY4NhfPX+23Gof9QU9ap5cwEAP7Pwp8xFYkugCSORC9gSaLJcvyG+KJw4H8GdLQ0Ji4O+0VofrLI8jXh9amK6IfEDFPIc43dvzsj0mE3f07NK9ArNHe3NiMZm0N3ZlpApon9+e68VA6PwpjcU0VrsGuJ8x/KFmD93DtqXLjAXid/eeRuW1VdjR3uz4xOAOl+fiKSup356qU51Em23FgKElBIU8jIm2zCPvRBIH9KsUhMfu2+d4zBnPWsFmBVVldutMle23mbM61YifeDYsLkB2lh3i2URef7okJnVcuCBIFqb6szr3tnSYF5Ht1+JuHpdFRs5hVycvy9p+0lI6cGslTImnayWVNiHND++q93x2vaMD3tGicojb6wzPP93R8bNrBfdXqdMlD3BFWhtqsVQeApPvtJvuf7Rc1dwZCAcr+o0ULNLn3r1LF4+9REAYCIaM58CVKaNwmkc3e6OZdjU1ojrsZueqlq9ZMqUSu90Uj7QIy8D3OK4uQzzOOVq6yPb9GIe/Xi7nfZwjBGTn0Fk8hP0HB9JyJTRaaitwoEHgubYOivW3iz6AOqWxhqcOB/B28PjODZolOxvalucYKPTOLpD/aPoDUXQG4pg8MrHKXPwvTwFldKGOCkPKORlQC6FIdNFQe9jXlM1DxPRWMLmogrL3NvaiOs3bqJ6/px4jvhcrcjnKp7e22GeY7ejtakOf/XrdyXYu7vjVjPDRp17ZCCMhtr5GIlEMRIx0hUDS+rQGxrDhlWLEuaZOrWzVQtNX/zJIVWv81QbzMZiNo3uzkBJbIiT8oBCXgbkMlPG60Bot1h4NDZjVpCqzUcAZndCVZRk5JPPluhHY9M4eeGqKZaq62CqlrJOvV/UZqyyYVNbIzasajAHZKjYO5A4z1SPxUdjM6ipmovdHbcCkAi2NKT8jlMteMbmaQjdnQFmwZCcQSEvA/IVQtGxe9zKuwasgq884x3tzbhnzahj69ue48OmR642Obs2rkbXxlkvfHZOqEywQd3TybvVj9ELoHTR1O01QjT9eHh7wOKFA0bapL4odXcGkqYiOi00ia9x85TkHgo5sZCqWZTyuLs7A5bNTidPXs/d1rNfaqrmoWujIWzK+1XnKRG/fuMmujvb0LVxdYIN+oJgeLdtpnDaY/lK8PWOi8peY/PUCNXYM2Jmh2sYlanrl3+EvpFx80nCaeScisnr79u/F71VLiG5gkJOPOE0OUj3Sp2qKJ3E3T75R2+qpbxX5el3d7ZhImoI5OO72tHaVGcT0MThE24Nw/RGW+uXL0oYFrEnuMJs1vXkK/2WOHx9jTF79K2hSEKHRf0eTh0Y7YuP3+sMSGlCISeeSVbl6FRF6ZzBYojuyQsTlqZaT716FqcvXcXD2wNY1VCDC+NRACIhhq3b0LWxxbLB6YZ9w3L98oUJqZMNtVX4+ufuwJe+/R4e3h4AYF2IUnVYdHvfSbhZLUpyDYW8iPjpH7Rb6MCedrijvdnxOJUd8vD2AE5fumrJANkTXGHZHFUNsbo2tmAiGgMwm26Yapanbo+aFNS1sQWP7Lwt5ff97sg4hsJTeHdkHC2Lay3x91SedDqeNtMPSa6hkBeRUvkH7WVBcQsd2D/D80eHHI/TS/Gf3tuBnuPDZjhF78xo3yQ91D+Kx3e141C/4em7zfK0z9h84+wVvD1sDKfQG3+5fcah8CSODYaxb/May2L02H3rAMCxyVeq7y2dwR+EZAOFvIiUyj9oLwuKW+jA/hmcjhufiiHQvACx6Zt4fFe7JZxSUzXX0vNcFz61AamKeKKxGTOFUA1IVqmQSnjVjM03zhpVmy2NNa5PCTpPfO999IaMFrvfeucDHBkI466W+qTnpkrVdBvczDg5yTUU8iJSKv+gvSwobrZ6aQGrpgs9dt86tDbVud7TLozqvcjkJ+gNjQGQZgqf2hB9ZOday9OCykqJTMYwf+4cfHHnWvNpwP6UoHvM7UsXoDc0hjWLa/FavHS/at5cM/a/qa0RkclP8Ozr57C741Yc6h+1hJN0j90+uDlZ215CcgGFnOR9QXESbXs3RT1/e09whUVkAaC6ah6UF67CKtdv3DSLf9R5ypM/cMxYOFQvl5bGGgSWzLYUAKwLx0OfaUN11VycvDCBkUgUrU21+Or9P4f6GkN4de9axfiB2XCS0wJkf7JQ7xOSayjkJGdkUt5v98LVcbPVlUYBkvLCr8dm0H/5J0b3xPlzzIKdx3e1m/e2LxxqI/XAMaPIaPDKpFmSH41N48OJ63jof/dBCIG3h8fNoRUqbAMAW29rwj8PjUEIgYe3B3DPmkZbv5hpS8xf/7ylEkIj5UtOhFwIMQLgYwAzAKallMFcXJeUFqk299Id8AwgwQvvOT4CQGJ3xzIAswVIm9oWo7uzDScvXEVvKILL1/4VBx4I4vSla2Z64pGBsFlWr9tobK4a1zXOH4NKZwQEvnnigmmPLuL6Z9q2tgnvjEwAAL7xxqBl4TCOFabH3rWxxfJeuimIfspmIqVBLj3ybVLKsRxej5QQbumHOnoaYarmUgpd5J4/OmSKoap+7Dk+jE1ti9EbGsPmwGJ89f6fw6+98A6GwlPoeWsYgSV1uDEjTS9ZldUrT16JoerXojfJMjBK5VfUV+O+f7MUDzm0zgUQr+78ECfOG6GaGzPvozc0Zi5aeul9qk3QHe3Nliwe+zGpvmdC7DC0Qjzhln6YyYBnp3MBo7fJvi1GyESJ+P7DIezbvBqbA4ltZ48MhHFx4joAYHNgMR7c2oqh8CROX7qGieiNeL75jMVDtndP1EvmnbxfPZav0750AebPFeai5VR679avRi12TlWi9u852TBsL98rPfrKIFdCLgG8JoSQAJ6XUh6wHyCE2AdgHwCsXLkyR7clhcIt/TCTAc9O5xqbiSHs27ImHhMH1KZmdXxMnPJWlXhvW7cE9TXzAQhT9Iw0wjHcvdqYGPSdH17CifNjeHt4wrTRC3Yx1FsHbFvbhIc+02YWK+1ob0743Mn61ei58nahddooVfdVqZqpKJX6BFI4ciXk90opPxJCLAHwuhDirJTyTf2AuLgfAIBgMMjWbz4jVTOtVB64k5eoSuejsWlcj80AAN7/8JrZmEovwddDDpvaFmPDqkXo2rja0lI3GpuOx7+BO5YvxNjkJxgKT+HixPWkPVKc4vpOqZBqKLXyjJX3rA+isD9l2GPleq48kFho5LRRqu7rdbOUm6uVR06EXEr5UfznFSHE3wO4C8Cbyc8ipUC2j+FePXAnL9EoDJpr6aa4o73ZzNG2FwepkIN9I1Vde1PbYnMWaNfG1fiPd63EI986hZGxKfyHTy2ztM/91Ip6fO37/fj9X2w3e6Tr8euHt892d1Tfkd45UQ2f3rd5DaKxaTNbZbbd70xCmqL9ewCch0Dbv181DDvX/01I+ZC1kAshagHMkVJ+HP/95wH8YdaWkYJQqMdwNy/RHkpQseiXT31kqYq0n697svomq4qVA4ag/eT6DfzkX6fxlZffx0T0hnnfhtr5GJ+6ga99vx8HHghChUj0TUgVS9fzwAGY7W6HwlPmIlATD//o8f5U3RDdhkATki65GL7cDKBXCPEegHcAfF9K+Y85uC4pAHuCuRvQrGMfMOw0TNnp9dmFRVrs0sMSPceHLQOeAWD98kXmLFD93lsCiwEAn1qxEItq5uH+O25Fd2cbvvrvfw4NtfPx+7/YblZvqr4u97Y2ItC8wLR9R3szNrUtRmTyE3NAsxLxh7cHLAObG2qr4i19x3FXS72ZHfP80SEMhSctYZXf/c6PcGQgjG+9M/s5vHyXhNjJ2iOXUp4HcEcObCFFIF+P4ak8fadYsgpXqIESykNX3vds6bs17KE3uLJXUv72zrVYVl+DF9/5AFej0zjz0TXs//yn4sJ4AwOjHyc8FWy5rQlPvXoW7394DcGWBlyPzaA3ZPR7qa6ah/XLF2L98kXo2tiCnuMj8da4H5kpjk98732zYdeheLm/nq1y4nwE65cvMo/pv3wt7e+SmSlEh+mHJC+k2nDTR8epoh4ldEqQ9ePUtaKxGQDSFLBnXx/AkYEw7m1tNHua2Ev2H9zaijtbGvClb7+H3//FdvMYVY0JIGGDUdny1lAEm9oaAQD3tjZCVZhuW9sUPzpxdJvq23JXS70WapnB9dgMbszcjBcuTePTKxdh/lyBr95/O4D0uiUWMjOFi0bpQyEneSGVp68LnIolP76rHYHmi3jzXBg72pvR2lSX4C2rzVGVs33ywlUAgBAiIa6u8+lV9Tj86GdMb/2f3r+M4bEoJqI3HNvcPr23A8/9IIT+yx/jiztvw+bAuGmLWngO9l3E7o5lOH3pmlmJChh9W9SAZyV8NVVzsf/wIFbUV2P5omozHfKx+9ahvsZ4iohMfoIDx4bNXuvJvstCZqYwnbH0oZCTvOPk0SlxMgp4rpqj3AZHP8ZbQxF85btnsOU2Y6PQrW/Jwb6L6A2NobWpFr2hMbQvXYBta5twZ0sDnn39HK7HZswMFj3lUR9i0VA737FVbUNtFRrrbkFvaNiygQrMDmxWGTb2FEQ34VUbpACwqa0RG1Y1WEJGm9oWx4+29lp3wmtILBfeNNMZSx8KeQVQyEdjp3sl68FiF0IlkoHmBfFe5GFsWNVg9iG3V4J2dwbMtrLKuwdg/gRgZpQAML3tP399AG8OjuGZvR2mDXqGyfhUzDIhSEe32UnkVDXmxNQNDIUn8ejPr8W7I+P4+ufuwJ/841ncmLmJ9qU/beaj64VCxlQjmOmM2UJvujKgkFcAhfzH7HSvZD1Y7EKoSujHp2IYHP0YRwbC6A1FUFM1FwAsaYD7D4fMHuetW+viWR0S12M3EWheEA9bS7MrobJvR3szltXX4Du/dS8aaquwsGa+2UFRX3z2Hw7F4+PSsvmaagScXgUKAP/y7fcwFJ7CY/etw/Z1S/DUq2fxww+uorHuloShGqr5llMVZ7JiIzdy4U1n0gyNFBYKeQVQyEdjp3sl68GSbGCFGglnr2p0+11NHtp/+Cweu2+dpVe4vhDooZUHt7aaHraaDXqofxR3tjTEQzaR+EJiePVKpLs7A66iuie4Am+eMzZKWxpr8PXP3YF3R8bN3PFjg2G0L13ouHm5qa3RTKO0k24xUbLvNx2SLcTJ4CZp4aCQVwCFrPTzOknI67XsVY1e+5m4/QSAO1uMPix6dsuxwTEcGQhjeOxdjESiZq743asbMG+OsBwLGFOL9h8eNLss2vvNfONXP20RsU+vqsf4VAxPvtKP3lAE8+daSzisRU1NjsKntzRQm6vZLs5exDbdZmgKhnUKRy4KgggpOG5FMvYCI/3v6vej567gyEAYX/nuGTMWvWHVIgDASCSKe1sbzUKftc115uarPjSiOt7Y6+SFq9jR3mwpXhoKT+LRl07hzpYGHOy7iKHwJJ4/OoSe48M4MhBGa1MtjgyE0XN8xPwMSiydirPUZwVU9ksIh/pHHQus0kWJrV5c5YT9e/VSpJSvYjOSCD1yUnJ48RKz8/aMrJC3hiJmqKBr42qcvDCB3lAEwZYGy4g4dWzP8WEz7bFrY4vZS+XJV/otsWPVgfHUxauYiN4wPW01HGN3x7L45qzqnW4Mw7CX6jsNcc51mCzT63n5/tnzpXBQyEnJ4aWS8c6WBqxqqMGHV6+nneGhMmBUYRGAeEtaYN/mNeja2IKJaMxMFzRSBesBCItdT+/tMNMW9dixKgiaiN4w8+PVBKPNgcWWzdmaqnmmoNtj9z3HR7D/8CAeuGeV2QYg1+KY6fWYklhaUMhJSeGW9mcX92+8MYgL41F8858vYNmiarNfeTJPXp8OpMrp1T33fbMPQ+EpzJ87x+xiqPqp6PM7VfEQ4Bw7Hp+KobpqHvZtXo3qqnlmiuHTezvw3NEhvHkujDvjHr/KkVeCrvcpNzCqRc+PTaI3FLHkqjt9b4XcWKS3XVpQyElJobJC9DJ9INEDfHxXO27MnLFkf6R63H/ie2fQG4pgeOxdM/VQnTcUnkJrU63Z5MpeUerGyNgUXuq7iDtbGrQBFINm1ozO4R+PYig8ZaYjKjt1UVSe+vNHh7C7Y5kp8Kq1r71/uf69sR9L5UIhJyWFLqB2IdKFsbWpDuuiV6QAABUDSURBVP/nv9zjeq6d8akYbswYHu5IJGoJhTiJtv1+enMufcDzl+Ki/KVvv4fDj37G1QZ9sdDTEZ1yw/WYuC7wKpXSPo/U7bOr0Ew0NmN5AiHlB4WclBT2Ycy5Sl872HcRbw+P497WRgRbGtKK7Q6FJ83Qi9EsS5qblPe2NmLmpsTXP3eHxX7lVeveNDC7WHx6VT0A4NnXB7D/cAiRyRgGrxgFUN2dbejuDCAyGcOzrw+YxUh6fxqnIR2J31FiQy9SnlDIScmS7oZastCK/Vr2eZx69og9FPHkK/2mN/303g4AMDcpv3niAzx23zozT7zn+AiUcO4/HMKL73yAkUgUb5y9gnvWNDhYbWTQ9F++ht5QBNvWNqFr42ozRKPupYdgVH+aHe3NruETIy1QmC2BSXlDISclS7obasmEP5mnr/ducVoIVNbJw9tnqzn1TUp9cVDie/fqBrQ01mAkEgUAvD08bvYf14uc1FxS5bnr1aF6y17F+FQMX/nuGbw1FMH65R+axUh6CqNKbXTaayDliZCy8I9dwWBQ9vX1Ffy+hADum4D21+1/VwuAKnKxX0N55H0j4+YAaSXmK+qrcXHiOh74t6twcTxqdntM1059Ebp7dQPuWL4QEMIcWq1Gz3V3BhKeLrxsfnKDtLQRQpyUUgbtr9MjJ74klegmw2sbAXuoxt5C1ykc88jO2zA+FcNzPxhC/+Vr+OJOo/Oh8pSPDRpNwIB+cyao0+cBZjcrI1MxNMa99D3BFYhMfoLX+kdNL39TW6Mp4o/vajdTGO0Crrx09XmcYFm9P6GQk4KTC6/PPjnI3k88F/bd2dKQMI/Tnumih2N0L72xrgq9oQg2B8YtmSc72pvxxPfOILBkgaWQSX2eF9/5AFsCi1Ffewuux6YBAK+9/y9miObBra1orLsFI5Go6e23L12IzYEm8/vUUxj1RUcfkedGrgp96NkXFgo5KTi58Pp0wVE9TO5tbcxJpaGyT4Up9EIcJVBqvuj12E2zW2HP8WHsPxzCscGwOb5NLxRS5wHAgWPnUV0110wL3BOcHTwxEvkAALBvyxq0NtWa2TL2UXb2uLpCT5W0f1den1b0hcCLENuFm559YaGQk4KTKt87/TiukfkRjBflZIJ+TWWXqrTUC3F0sTZCJNDmdxp29IYiCf1X9J7e6rzrsWmLWB54IIjf/Guj++KmtsWonj/HbBGwfvlCc5SdyiOvr6myPAXo91LNuVKV9bt93+kKcbIwFMk/FHJScJIJixcBsR+jMj8yFQ27B/vg1lZLOOTZ18+ZhTVKrKOxGdy9ugFSSm1+5604eWHcHLCsN9nSF4fAkg/w3qVreO/SVXN254NbW9HaVIfv/Na9prBORGM4fekaAksWxHugt+Gx+9YhMhXD/sOz8flkgzwO9buX9Sf7vtMVYvvxLOEvLBRyUlJ4ERAjNW/anPyTrWgoD1Yf9ab3ZdELa7o2rja7HgJAd2cAnT/bbHrGvaEIujsD2L6uOaFwR9nYWHeLmYqo3xMwmnedOB+xzAO9MSPNrokvn/oQ712cMO1x+r7cesA4ed5u37fTd5rsacmpEtZLjJyx9NxAISdFxf4P2Ysoq0lAavJPtkOInWLIT77Sb04Nenpvh6Uk3phcNAJD2Fscy+Sdmmzp91M54rs7llnsst93dtjEYhzqHzWzTlThUKoMHBXrdir7dzoumaCmE27xeixj6TlCSlnwPxs2bJCESCnlcz8IyVVffkU+94NQWudFJj+Rz/0gJCOTn6R1n2deG/B0XujKx/LXXnhbhq58nJENTu85vWb//CdHxuX2PzsiT46Mm+c889qAfOa1szJ05WP5zGtn5TOvDcjI5CeW67nZMvu5zyY91u2/g5d7pPvdZHIcMQDQJx00lR45KSqZboplWvXpVr1pRw2BTkYyb9LpPT333AjXCGy9rcmS4nj03BUMhafwJ//4Y9yzphFdG1ebcfBXTl82G25NRGOa926QKtatnhLs+wFG6+AZx1mh9s/h1Xv3+t+HsfTcQCEnRaVQ/5D1EIJeVu8Vp9BMskXI6b0d7c04cT6C67FpHDg2DADxqURjuGfNKOqDVThx3oidvz08gbeHJ0xbVWqi6raoQi72GLtbrFsPsdjP01vvJgs7KZwWJD3ERAoPhZyUDV4HCWeycNg901T3crqP2rxcv3wR7m01qjHbly7A5sBic7NUbYLevboB96xpsKQmPvmK0e9FVYraKzidUhHt9qvMlx3tzeaxaoGxj5pz+xz6042K2bvtVXAzszBQyEnZkM+NM7tnmsm99Gt0bWwxC4QO9Y+ar0dj07h+4yaq588xN0LVMSovvWVxbdq53/YQi96zBUBC4VMydC/fQLg+4XAzszBQyEnZkE68PVtPMZPYvi6A6t52oXtk51qzR3nfyATeGopYZnnq7QhOnI9Yio7S8azt9tvTOXXcviujt8xazxlBJH/kRMiFEJ8FsB/AXAD/S0r5x7m4LiHpkE7YJNvKxVT3MjohDkOPH882r5pNBXQWOqPoaPqmRHdnALs7bjXDKCrnvaF2fsLQZxW6sXvWTkKr2z8+ZRQeHRkIO4ZIUn1Xyd7nZmZhyFrIhRBzAfwPADsBXALwrhDiZSllf7bXJiRfePEUncr2kxXY2D1te/zYHqdW59qFrmtji1l0tH3dErQ21ZmNsKKxGWxqazSHUOj239nSgNamWqxtXmDJKkk18s1eEGX/bKm+K3rdxScXHvldAEJSyvMAIIT4WwD3A6CQk5LFi6folHrn9p79NRXvVvFjPcVPFfIks81emamuv//wIFoaa7Bvyxo8pA3K2BNcgW+8MYih8BT+4P+9j5FIFNHYdHyIhXXkm12od7Q3481zYQSaF1g+hx66SVXlmUmjLZI7ciHkywBc1P5+CcDd9oOEEPsA7AOAlStX5uC2hOSXTNILjw2OITL5CSaiMUs16PNHh1xT/JxwyxZRaYiDox8nbFo+vD2AD8aj2LCyPt721gjRGLno88zmX/Yqz0P9o3hrKIK3hiKonj8HgMC9rY1m6EbPhgGAnuPDOHnhKnpDY+Y1AG5sFpNcCLlweC1h7JCU8gCAA4AxISgH9yUkr6Qb3z3UP4re0Bh6Q2MYvDJpKbrRRd5pQ9ErneuasXThtXgPGOuCcjAu8rvW32rpO64+hxJ9PbSjzlVPDwCw//AgujsD2HJbU8KGrLEIzLYJcMphZ4il8ORCyC8B0P/LLQfwUQ6uS0jJ4uR92nuoqA1KwCryjXW3WDYanWLtasLQV++/3RwJd7DvIg4cO4/H7ltnGRMXjU2j5/gIdnfcCgCWlEYdp54yswjTbvuIOP08YwMXWNVQg8d3tSdksSRrlzvbnyZ5aImkTy6E/F0AASHEagAfAvgVAL+ag+sSUrK4dR3UNxP1zBFd5J2qJAFriOLAsfMAjOZdf/XrdyWU0avujIEldWaVqNpUtQ+X1hcLt6wTFWqpqZrnuMio14zuj9c8tcj1cg+SG7IWcinltBDiCwD+CUb64QtSyveztoyQEiadYQ3q705l7E4Lwp7gCkQmY+i/PBtCsZfRq1xy1eL2+o2biMamMT4Vcy1esued26cdJVtk9Di500ZsKtwWMpIjnDpp5fsPux8SP+K1U5+9k2CmHR6T3dvenTHZPSKTn8hfe+HthPdT2aXf85nXzppdFNV7eldGdjAsDGD3Q0Kyw2tWht0jTpWDnkmPmNamOtMzrg9WmR6v8srt11m/fCHWL1/kuDmpj7Jzj3kLy089VKJCLam+F/ZdyR8UckI84jUrwy66qqjGrSzfLfThhlOF6INbW812t/YYtCpO0lMfk7UJsL/fUFuF3R234vSlq+aGqlo4rsfDJfZFwgmmJ+YPCjkhHsmm3Nweb7b/VP1U9JJ7N9QA6H2bV1vSCN16rTg9EagYu4pbd3cGHN8HZnPN9fJ/tbGrNla95MczPTF/UMgJKQD21D97kc365YuwfvlCjyJnhDeqbZ63W68V+wKkl+Rfj83gwLHz2Ld5jcVDt/csV7nm9sZa6bY6oCeeHyjkhOQJpzJ2he6hA0io+kwWT+7a2OI6C1T/6YZTXnj/5WvoDUUcr6Pi52pO6ulLV80QkL35lr1RmP2z5lPIKzkGTyEnJE8kEzA9DFJfY/VuU52bbGK91/FqyvPeetsSnL50DQ9vD2BzYDxh0dFz0lOFgJwahemfK98hlUqOwVPICckTbgI2PjU7b/OeNaMJDbmSneuEl81Kt3O2rW2K29HoeK49JJQsh9zeKEyRTs69F9vdqOQYPIWckDzhJmBOMehk57oJm17QAyT36N1a0+oj49zOtWfguImyGjThhG7rof7ZEXVORUf6CLl0POtK7n1OISekwCTveTKLW5qhwutoN6djddGzl9nnyrN1SnHUpx09uLXVsqCoTJnuzoAlG8ftmpUWB08GhZyQAuPVc3QaRKGTTHBTjXZzItfZJU792a/fuGnJztE7M6qnFKdWBm7XpKgbUMgJyQO58BydPPdkmTDJyGSQRjJSfT57k6+G2irUVM3D/sPOOedumTLJjqvkzU07FHJC8kAuRMZJfDO5bjLRTTbOLhnKjmhsJqHtLTBbtNTd2eYp59wtUyZZxk4lb27aoZATkgfyJTJeQyR6Pncy8U+2senFjmhs2uXawvbTe0gp01YIlQyFnJA8kC+RsVeFuqUW6vncyZpj6fns6YSD9DmdaqSdjlvRktfP6JYnX+mxcDco5IT4jFQbfvZ8bvuoN2DWe1Zl/euXf4TTl6566mKo47Zg5XIhYyw8NRRyQnxGqg0/t3xup5CFHiJJldteLNIJU1Wq904hJ8QnOKUHpiNyTl6yU4jEa4/0XJLsful495Xqvc8ptgGEEG8okTrYd9F8TYmcEt/njw5hfCqW9rX167jdKxe42Zir++0JrnAtJipn6JET4hNSed/ZeKP2Enqnsv9srqXK8t1sZCphdlDICfEJqUIM2YihWwl9JtivpV/TzcZcbY5WamiFQk5ImZCNGDo10cq0M6H9Wvo19Zi8W/VmNlSqZ08hJ4Q4NtFySlf04vE6XcvemCtfnrPXxazcslso5IQQR5KlK2br8SYrUkpGrgS43EIwFHJCiCPJ0hUzxZ5C6eT1JxPrVD1evFJuIRgKOSEVTiHDDPpG6NN7OxwF1UufdfceL94otz4tFHJCKpxChhn2BBPnfqYz5i5Vj5dKRUgpC37TYDAo+/r6Cn5fQkgipVTFWWzbSh0hxEkpZdD+Ois7CalA9ApLe1Wn23GZXNuJZPezo54Weo6PWK6ZTRVrOUIhJ6QC8VoSr4579KVTnkUzl+X9quQekJZr5quFgF/JKkYuhPgDAP8VQDj+0u9KKf8hW6MIIfnFa9aGU0w7V9f2gltMvNyyTrIlqxh5XMgnpZR/ls55jJET4h+8jopjDDv/MEZOCMmIZDFthjhKg1ykH35BCPEAgD4Aj0opJ3JwTUKID2CIozRIGVoRQhwC8DMOb/0egBMAxgBIAF8DsFRK+Rsu19kHYB8ArFy5csOFCxeyMJsQQioPt9BKzvLIhRAtAF6RUt6e6ljGyAmpDLKNodt7m1d6LN5NyLPNWlkqpbwc/+svAziTzfUIIeVFtlWjueyTXs5kGyP/UyFEB4zQygiAB7O2iBBSNmQbQ3fqk04SYYk+IYT4BKYfEkJImUIhJ4QQn0MhJ4QQn0MhJ4QQn0MhJ4QQn0MhJ4QQn0MhJ4QQn0MhJ4QQn0MhJ4QQn0MhJ4TkDM7SLA4UckJIzuCgCXfyucjlYrAEIYQA4KCJZGTbCTIZFHJCSM5QY+FIIvlc5CjkhBBSAPK5yDFGTgghPodCTgghPodCTgghPodCTgghPodCTgghPodCTgghPodCTgghPkdIKQt/UyHCAC4U+LaLAYwV+J65wq+2+9VuwL+2+9VugLZ7YZWUssn+YlGEvBgIIfqklMFi25EJfrXdr3YD/rXdr3YDtD0bGFohhBCfQyEnhBCfU0lCfqDYBmSBX233q92Af233q90Abc+YiomRE0JIuVJJHjkhhJQlFHJCCPE5FSXkQoivCSFOCyFOCSFeE0LcWmybvCCE+LoQ4mzc9r8XQiwqtk1eEULsEUK8L4S4KYQo+dQyIcRnhRADQoiQEOJ3im2PV4QQLwghrgghzhTblnQRQqwQQhwRQvw4/v9Kd7Ft8oIQ4qeEEO8IId6L2/3VotlSSTFyIcRPSyl/Ev/9vwFol1I+VGSzUiKE+HkAb0gpp4UQfwIAUsovF9ksTwghfhbATQDPA/jvUsq+IpvkihBiLoBzAHYCuATgXQCfl1L2F9UwDwghtgCYBPBNKeXtxbYnHYQQSwEslVL+UAixAMBJAL9U6t+7EEIAqJVSTgoh5gPoBdAtpTxRaFsqyiNXIh6nFoAvVjEp5WtSyun4X08AWF5Me9JBSvljKeVAse3wyF0AQlLK81LKGIC/BXB/kW3yhJTyTQDjxbYjE6SUl6WUP4z//jGAHwNYVlyrUiMNJuN/nR//UxRNqSghBwAhxB8JIS4C+E8AvlJsezLgNwC8WmwjypRlAPTx75fgA0EpJ4QQLQA+BeDt4lriDSHEXCHEKQBXALwupSyK3WUn5EKIQ0KIMw5/7gcAKeXvSSlXAPgbAF8orrWzpLI7fszvAZiGYXvJ4MV2nyAcXvPFU1s5IISoA/B3AH7b9vRcskgpZ6SUHTCeku8SQhQlrFV2w5ellDs8Hvp/AXwfwBN5NMczqewWQnQB2AWgU5bYxkYa33mpcwmAPuJ8OYCPimRLRRGPMf8dgL+RUn6n2Paki5TyqhDiBwA+C6DgG85l55EnQwgR0P66G8DZYtmSDkKIzwL4MoDdUspose0pY94FEBBCrBZCVAH4FQAvF9mmsie+afiXAH4spXym2PZ4RQjRpDLIhBDVAHagSJpSaVkrfwdgLYwsigsAHpJSflhcq1IjhAgBuAVAJP7SCT9k2wCAEOKXAXwDQBOAqwBOSSn/XXGtckcI8QsA/hzAXAAvSCn/qMgmeUII8SKAz8BopzoK4Akp5V8W1SiPCCE2ATgG4Ecw/m0CwO9KKf+heFalRgixHkAPjP9X5gB4SUr5h0WxpZKEnBBCypGKCq0QQkg5QiEnhBCfQyEnhBCfQyEnhBCfQyEnhBCfQyEnhBCfQyEnhBCf8/8BA0AxVAlHNBoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(features[:, 1], labels, 1);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_iivzo2j",
    "id": "27981A0FD4054AC39194415A90F313EC",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 读取数据集返回批量数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T16:04:04.151776Z",
     "start_time": "2020-03-26T16:04:04.147001Z"
    },
    "graffitiCellId": "id_0tj7eus",
    "id": "A6E1419DA00C4ABF8CBF0E0F0B2B9E35",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def data_iter(batch_size, features, labels):\n",
    "    num_examples = len(features)\n",
    "    indices = list(range(num_examples))\n",
    "    # 样本读取顺序随机，会遍历所有样本\n",
    "    random.shuffle(indices)\n",
    "    # 每次取 batch_size 的数据\n",
    "    for i in range(0, num_examples, batch_size):\n",
    "        # j 为索引数组\n",
    "        j = indices[i: min(i+batch_size, num_examples)]\n",
    "        # axis表示维度，indices表示在axis维度上要取数据的索引\n",
    "        # tf.gather（在某一维度指定index）\n",
    "        # yield features[j], labels[j]\n",
    "        yield tf.gather(features, axis=0, indices=j), tf.gather(labels, axis=0, indices=j)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T16:04:04.786254Z",
     "start_time": "2020-03-26T16:04:04.780271Z"
    },
    "graffitiCellId": "id_xc0arq3",
    "id": "1DA3BC30E43E4F76970F712D89BDBC4D",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[ 0.15303102 -0.34378156]\n",
      " [-0.25278276  1.9003394 ]\n",
      " [ 1.4838715   0.98826146]\n",
      " [ 0.589835    0.3098992 ]\n",
      " [ 0.47091532  0.18669523]\n",
      " [ 0.24663025 -0.22130807]\n",
      " [ 1.0166736   1.6568373 ]\n",
      " [-0.46696392 -1.4222957 ]\n",
      " [ 1.2127599  -0.5779025 ]\n",
      " [-0.00413441 -0.6999474 ]], shape=(10, 2), dtype=float32) \n",
      " tf.Tensor(\n",
      "[ 5.674328   -2.7682092   3.8202336   4.316163    4.499334    5.4640665\n",
      "  0.59213424  8.106868    8.59062     6.573457  ], shape=(10,), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "batch_size = 10\n",
    "\n",
    "for X, y in data_iter(batch_size, features, labels):\n",
    "    print(X, '\\n', y)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_hj6sxxx",
    "id": "1FF819B45B1F44C88012EBB266C10EE8",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 初始化模型参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T16:04:06.405180Z",
     "start_time": "2020-03-26T16:04:06.400194Z"
    },
    "graffitiCellId": "id_g06bzki",
    "id": "6B11AC0E574140CD9C2E722B05D0049D",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 2行1列，正态分布随机数\n",
    "w = tf.Variable(tf.random.normal((num_inputs, 1), stddev=0.01))\n",
    "b = tf.Variable(tf.zeros((1,)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_zvsctyc",
    "id": "A91414B8FDF24835A06B6ADFAEC2C15C",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 定义模型\n",
    "定义用来训练参数的训练模型：\n",
    "\n",
    "$$\n",
    "\\mathrm{price} = w_{\\mathrm{area}} \\cdot \\mathrm{area} + w_{\\mathrm{age}} \\cdot \\mathrm{age} + b\n",
    "$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T16:04:08.560354Z",
     "start_time": "2020-03-26T16:04:08.556395Z"
    },
    "graffitiCellId": "id_l8xu5kf",
    "id": "8DFF5BDD78884936899E3CE720BEEE3C",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def linreg(X, w, b):\n",
    "    return tf.matmul(X, w) + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_1sta0nq",
    "id": "C9B747281D1842C682F2AEB1F38B959D",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 定义损失函数\n",
    "我们使用的是均方误差损失函数：\n",
    "$$\n",
    "l^{(i)}(\\mathbf{w}, b) = \\frac{1}{2} \\left(\\hat{y}^{(i)} - y^{(i)}\\right)^2,\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T16:04:09.682037Z",
     "start_time": "2020-03-26T16:04:09.679045Z"
    },
    "graffitiCellId": "id_r9p6ncn",
    "id": "58A55DD7B46842578BEA1A8689456B1A",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def squared_loss(y_hat, y):\n",
    "    return (y_hat - tf.reshape(y, y_hat.shape)) ** 2 / 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_jm7ie9i",
    "id": "0A98B83A8FFD4E84B6EFE8A894643634",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 定义优化函数\n",
    "在这里优化函数使用的是小批量随机梯度下降：\n",
    "\n",
    "$$\n",
    "(\\mathbf{w},b) \\leftarrow (\\mathbf{w},b) - \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}} \\partial_{(\\mathbf{w},b)} l^{(i)}(\\mathbf{w},b)\n",
    "$$\n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T16:04:10.761714Z",
     "start_time": "2020-03-26T16:04:10.757725Z"
    },
    "graffitiCellId": "id_e41t41x",
    "id": "E9676D1B4F80473B894A4ADA3691D2E0",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def sgd(params, lr, batch_size, grads):\n",
    "    \"\"\"\n",
    "    Mini-batch stochastic gradient descent.\n",
    "    lr: 步长\n",
    "    grads: tf 自动梯度计算等到的梯度和\n",
    "    \"\"\"\n",
    "    # 对每一个参数求梯度，并更新\n",
    "    for i, param in enumerate(params):\n",
    "        # grads 除以批量大小，得到平均值\n",
    "        param.assign_sub(lr * grads[i] / batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_0nsokgo",
    "id": "B18F2D19AA1140478E2E327ECC97F40F",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 训练\n",
    "当数据集、模型、损失函数和优化函数定义完了之后就可来准备进行模型的训练了。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T15:35:10.192347Z",
     "start_time": "2020-03-26T15:35:09.798351Z"
    },
    "graffitiCellId": "id_ht68g0d",
    "id": "8C7AA862EE5A4AEAB3CB980F15870D06",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 1, loss 0.036769\n",
      "epoch 2, loss 0.000132\n",
      "epoch 3, loss 0.000050\n"
     ]
    }
   ],
   "source": [
    "lr = 0.03\n",
    "num_epochs = 3\n",
    "net = linreg\n",
    "loss = squared_loss\n",
    "\n",
    "for epoch in range(num_epochs):\n",
    "    for X, y in data_iter(batch_size, features, labels):\n",
    "        with tf.GradientTape() as t:\n",
    "            t.watch([w, b])\n",
    "            #print(net(X, w, b))\n",
    "            #print(y)\n",
    "            l = loss(net(X, w, b), y)\n",
    "            l = tf.reduce_sum(l)\n",
    "        # 求 w ，b 梯度\n",
    "        grads = t.gradient(l, [w,b])\n",
    "        # 梯度下降, 更新参数\n",
    "        sgd([w, b], lr, batch_size, grads)       \n",
    "    train_l = loss(net(features, w, b), labels)\n",
    "    print('epoch %d, loss %f' % (epoch + 1, tf.reduce_mean(train_l)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T15:38:55.785867Z",
     "start_time": "2020-03-26T15:38:55.780851Z"
    },
    "graffitiCellId": "id_6t702dg",
    "id": "2E791A3F92EF4CCF91E2096630C0E8D9",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2, -3.4] \n",
      " <tf.Variable 'Variable:0' shape=(2, 1) dtype=float32, numpy=\n",
      "array([[ 1.9997437],\n",
      "       [-3.3994198]], dtype=float32)>\n",
      "4.2 \n",
      " <tf.Variable 'Variable:0' shape=(1,) dtype=float32, numpy=array([4.1998153], dtype=float32)>\n"
     ]
    }
   ],
   "source": [
    "print(true_w, '\\n', w)\n",
    "print(true_b, '\\n', b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_pi6pxp6",
    "id": "7E8D79B69557446883330AB1E8DE07E2",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 线性回归模型的简洁实现\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_07nlorv",
    "id": "34B9AE6FB3D64DFD83E93D5CEF9EEE65",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 生成数据集\n",
    "在这里生成数据集跟从零开始的实现中是完全一样的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "graffitiCellId": "id_k7z5rd0",
    "id": "83C2DB9468394624BB4934DBF194A353",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "num_inputs = 2\n",
    "num_examples = 1000\n",
    "true_w = [2, -3.4]\n",
    "true_b = 4.2\n",
    "features = tf.random.normal(shape=(num_examples, num_inputs),stddev=1)\n",
    "labels = true_w[0] * features[:, 0] + true_w[1] * features[:, 1] + true_b\n",
    "# labels 增加随机干扰\n",
    "labels += tf.random.normal(labels.shape, stddev=0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_io6yz0p",
    "id": "0FB74CD3CD784A82B2A422E54BB0DEDD",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 读取数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T15:41:30.322022Z",
     "start_time": "2020-03-26T15:41:29.480691Z"
    },
    "graffitiCellId": "id_bxmqh9f",
    "id": "8704CA375BF04440839AB16AA995E3AB",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tensorflow import data as tfdata\n",
    "\n",
    "batch_size = 10\n",
    "# 将训练集的特征和标签组合\n",
    "dataset = tfdata.Dataset.from_tensor_slices((features, labels))\n",
    "# 随机读取小批量数据\n",
    "dataset = dataset.shuffle(buffer_size=num_examples)\n",
    "dataset = dataset.batch(batch_size)\n",
    "# iter 方法创建一个迭代器\n",
    "data_iter = iter(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T15:41:33.980346Z",
     "start_time": "2020-03-26T15:41:33.925457Z"
    },
    "graffitiCellId": "id_nnjw15x",
    "id": "C1FFC0FD8F5741D78AFD26B883BE192C",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[-0.1669998   0.44764984]\n",
      " [-0.08119935 -0.96780413]\n",
      " [-0.57150066 -0.7719588 ]\n",
      " [ 0.7352289  -0.17188445]\n",
      " [ 0.26945013 -0.6415312 ]\n",
      " [ 0.21700485  1.0177644 ]\n",
      " [-1.5488733   0.5636575 ]\n",
      " [-2.616565   -0.8470878 ]\n",
      " [ 0.23921391 -0.19487649]\n",
      " [-0.6440835  -1.2607629 ]], shape=(10, 2), dtype=float32) \n",
      " tf.Tensor(\n",
      "[ 2.3396387   7.330994    5.670597    6.2626505   6.9204926   1.1823338\n",
      " -0.81205755  1.8415534   5.333409    7.2056746 ], shape=(10,), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "for X, y in data_iter:\n",
    "    print(X, '\\n', y)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_zobpfwu",
    "id": "F9085AAAB3BB45E289329A5EA5446848",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 定义模型和初始化参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T15:43:28.953907Z",
     "start_time": "2020-03-26T15:43:28.923987Z"
    },
    "graffitiCellId": "id_gxy6vho",
    "id": "28DD8C6981314D148B5FD1915639151C",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "from tensorflow import initializers as init\n",
    "\n",
    "model = keras.Sequential()\n",
    "model.add(layers.Dense(1, kernel_initializer=init.RandomNormal(stddev=0.01)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_l729glu",
    "id": "BBFF587F757A4C7EB49AD0D536AD363E",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 定义损失函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T15:43:43.848574Z",
     "start_time": "2020-03-26T15:43:43.844586Z"
    },
    "graffitiCellId": "id_or1wah4",
    "id": "B721F8DD4811434BB1984B5B2DABC143",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tensorflow import losses\n",
    "loss = losses.MeanSquaredError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_zyt512e",
    "id": "6490FA20F3D4462CB2B98902F694E525",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 定义优化函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T15:43:54.890415Z",
     "start_time": "2020-03-26T15:43:54.886428Z"
    },
    "graffitiCellId": "id_pmx4gbq",
    "id": "1998CEB53B534F178AC6223011627B0B",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras import optimizers\n",
    "trainer = optimizers.SGD(learning_rate=0.03)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_n2klgfl",
    "id": "090AC5BD4E214B75BD7C4AB9B68720D0",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T16:03:08.593272Z",
     "start_time": "2020-03-26T16:03:07.681247Z"
    },
    "graffitiCellId": "id_qj2fl3l",
    "id": "A4B0F83F71F94728811A619F1AE74CD2",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 1, loss: 0.000215\n",
      "epoch 2, loss: 0.000100\n",
      "epoch 3, loss: 0.000101\n"
     ]
    }
   ],
   "source": [
    "num_epochs = 3\n",
    "for epoch in range(1, num_epochs + 1):\n",
    "    for (batch, (X, y)) in enumerate(dataset):\n",
    "        with tf.GradientTape() as tape:\n",
    "            l = loss(model(X, training=True), y)\n",
    "        grads = tape.gradient(l, model.trainable_variables)\n",
    "        trainer.apply_gradients(zip(grads, model.trainable_variables))\n",
    "        \n",
    "    l = loss(model(features), labels)\n",
    "    print('epoch %d, loss: %f' % (epoch, l))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-26T16:03:32.683678Z",
     "start_time": "2020-03-26T16:03:32.678692Z"
    },
    "graffitiCellId": "id_ke4hsr4",
    "id": "704087439A114181B3A7FE79539127AB",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2, -3.4] [[ 2.0008821]\n",
      " [-3.3996172]]\n",
      "4.2 [4.200015]\n"
     ]
    }
   ],
   "source": [
    "print(true_w, model.get_weights()[0])\n",
    "\n",
    "print(true_b, model.get_weights()[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_v7cg0i4",
    "id": "A968DC29635C4CDF8394A6F779661DC5",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 两种实现方式的比较\n",
    "1. 从零开始的实现（推荐用来学习）\n",
    "\n",
    "   能够更好的理解模型和神经网络底层的原理\n",
    "   \n",
    "\n",
    "2. 使用`Tensorflow`可以更简洁地实现模型。\n",
    "\n",
    "    `tensorflow.data`模块提供了有关数据处理的工具，`tensorflow.keras.layers`模块定义了大量神经网络的层，`tensorflow.initializers`模块定义了各种初始化方法，`tensorflow.optimizers`模块提供了模型的各种优化算法。\n",
    "   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "165px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
